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ERROR CONCEALMENT OF VIDEO DATA USING 
TEXTURE DATA RECOVERY 

BACKGROUND 

5 The present invention relates to error concealment, and 

more particularly, to concealing errors in texture partition 
of a video packet. 

In an MPEG-4 decoder used for real-time streaming, when 
an error is detected in the texture partition of a video 
10 packet, substantially all decoded texture information of that 
packet is often discarded. In video data, texture represents 
luminance and chrominance information of the pixels in the 
frame. The region of the video frame corresponding to this 
corrupted packet may then be generated by error correction 
15 process. However, traditional error detection and correction 
systems may require a significant amount of overhead as well 
as significant amount of data processing when decoding coded 
video bit stream signals. Thus, current video decoding 
systems may rely on error concealment techniques for 
20 effective video streaming. In contrast to error correction, 
which attempts to reconstruct lost or corrupt data, error 
concealment aims to generate data that may be substituted for 
the lost or corrupt data. Discrepancies in image created by 
the generated data (generally at the macroblock level) are 
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not likely to be perceived by a viewer of a video image that 
utilizes such error concealment . 

Motion compensated temporal replacement may be an 
appropriate error concealment technique in video frames 
5 without much scene changes. This technique uses motion 

information to propagate macroblocks of the texture data from 
the" previous frame to replace the texture data of the 
corrupted macroblocks in the current frame. Thus, this 
technique may provide effective error concealment as long as 
10 motion information can be obtained without much error. 

However, there are times when the texture data cannot be 
adequately concealed just by copying texture data from the 
previous frame at some motion compensated displacement . 
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SUMMARY 

In one aspect, a method for concealing errors in texture 
partition of a video packet is described. The method 
includes determining a particular macroblock within the 
5 texture partition where error is detected, concealing the 
error starting at the particular macroblock, and evaluating 
image smoothness of concealed macroblocks . The concealing 
and evaluating are repeated with one more macroblock added 
prior to the previous particular macroblock. The concealing 
10 and evaluating are repeated until all macroblocks in the 

texture partition have been concealed. A set of macroblocks, 
including a combination of decoded and concealed macroblocks, 
that produces best image smoothness is then selected to 
replace the corrupted texture partition. 

15 In another aspect, an error concealment system for 

texture partition of a video packet is disclosed. The system 
includes an error location detector, an error concealment 
element, an image smoothness evaluator, and a selector. The 
error location detector receives video packets, and 

20 determines a particular macroblock within the texture 

partition where error is detected. The error concealment 
element conceals the error starting at the particular 
macroblock. The image smoothness evaluator evaluates the 
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concealed macroblocks . The selector then selects a set of 
macroblocks, including a combination of decoded and concealed 
macroblocks, that produces best image smoothness. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figures lA through IB describe the present error 
concealment technique in accordance with an embodiment of the 
present invention. 

Figure 2 shows an example of a corrupted video packet 
with relevant parameters designated on the packet. 

Figure 3 illustrates an image smoothness test of texture 
data in accordance with an embodiment of the present 
invention. 

Figure 4 shows an error concealment system according to 
an embodiment of the present invention. 
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DETAILED DESCRIPTION 

In recognition of the above-stated difficulties with the 
prior error concealment techniques, the present invention 
describes embodiments for error concealment of video data 
5 using texture data recovery. In concealing corrupted video 
packets, the decoder utilizes uncorrupted received texture 
information. Macroblock boundary smoothness is evaluated for 
varying number of macroblocks of utilized texture 
information. The final number of macroblocks of utilized 

10 decoded texture information (the recovered information) is 
selected to maximize overall boundary smoothness. Although 
present embodiments are applicable to any video data, they 
are particularly well suited for MPEG- 4 decoders used for 
real-time streaming video. Consequently for purposes of 

15 illustration and not for purposes of limitation, the 

exemplary embodiments of the invention are described in a 
manner consistent with such use, though clearly the invention 
is not so limited. 

FIGS . lA through IB describe the present error 
20 concealment technique in accordance with an embodiment of the 
present invention. The technique is described in a flowchart 
form. 
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In the illustrated embodiment, determination is made (at 
100) as to the presence of any errors in the texture 
partition of a video packet. Detecting invalid variable 
length code (VLC) , inconsistent resynchronization header 
5 information, and/or receipt of out-of-range motion vectors 
and other related invalid codes may determine video packet 
errors. Since variable length codes are used to compress 
video data, the location where the decoder detects an error 
(e.g. where invalid VLC occur) may be some undetermined 

10 number of bits away from the actual location of the error. 

Accordingly, if the error is detected at 102, then the video 
packet may be further processed at 106. Otherwise if no 
error is detected at 102, then the next video packet may be 
processed at 104. FIG. 2 shows an example of a corrupted 

15 video packet 200. 

In video data encoded in Moving Picture Experts Group 
(MPEG) format, a video frame is divided into macroblocks . In 
this format, each video packet includes information for 
consecutive macroblocks from left to right and top to bottom. 
20 The number of macroblocks (N) represented by each video 

packet may be determined by the change in macroblock number 
field of the video packet header. Thus, packet processing 
(at 106) may include determining the location of the detected 
error 2 02 with respect to the total number of macroblocks (N) 
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204. This error location determination may be made as 
occurring at a particular macroblock {K) 206 from the end of 
the texture partition 210. Alternatively, the error location 
may be determined to occur at N-K macroblocks from the 
5 beginning of the texture partition 208. Therefore, an 
assumption may be made that the texture information is 
corrupted starting from the location of the detected error 
202. Accordingly, K is initialized to the number of 
macroblocks in the packet that was decoded after the first 
10 detected error of the packet. This initial value of K {i.e. 
Kinit) 206 is the least number of macroblocks that is 
concealed. 

Since N-Kinit is the largest number of macroblocks to be 
decoded but not concealed, the texture information of the 

15 first N-Kin±t decoded macroblocks is stored in a temporary 
buffer, at 108, to be used later in selecting a set of 
texture data with the best boundary smoothness measure. 
Further, the last K macroblocks of the texture data is error 
concealed, at 110. In the illustrated embodiment of FIG. lA, 

20 the error concealment process, at 110, includes motion 
compensated temporal replacement. This may involve 
concealing the corrupted texture data with motion compensated 
texture data from the previous frame using corresponding 
motion vectors. To save computation time, motion compensated 
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temporal replacement may only be required on macroblocks 
whose motion vector has changed. The first N-K macroblocks 
assumed to have non-corrupted texture information may be 
considered as "non-corrupted" in the above error concealment 
5 process. 

In the illustrated embodiment, the image smoothness of 
the corrupted video packet macroblocks restored by temporal 
replacement (at 110) may then be evaluated, at 112. The 
image smoothness may be measured by computing the smoothness 

10 of macroblock boundaries in the restored texture data. In 
one embodiment, this boundary smoothness is measured by 
summing the pixel value mismatch between macroblock boundary 
pixels. Thus, this method measures the image smoothness 
spatially (see FIG. 3) . This embodiment is described in 

15 detail below in conjunction with description of FIG. 3. 

The value of K that produces the best overall boundary 
matching (i.e. smallest sum of boundary pixel mismatch for 
all macroblocks of the corrupted video packet) is stored, at 
114, as Kbest for later use. Moreover, by storing the partial 
20 sum of the mismatch values (mismatch values for each 

macroblock) , the sum of the mismatch values in each iteration 
may be computed more efficiently by reusing the partial 
mismatch values from previous iteration for macroblocks that 
itself and its neighbors did not change. 
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At 116, the variable K is incremented by one, which 
moves the particular point of demarcation between the decoded 
and the concealed texture data further to the beginning 208 
of the texture partition in the video packet. If this 
5 particular point of demarcation has not reached the beginning 
of the motion partition 206 in the video packet (i.e. K^N), 
at 118, then the process of error concealing by creating 
motion compensated temporal replacement and evaluating the 
image smoothness/matching with one more macroblock of 

10 concealed texture data is repeated, starting at 110. 

Otherwise when K = N, all macroblocks of the texture data 
have been concealed. Thus when K = N, a. combination of 
recovered decoded texture data and concealed texture data 
that produced a lowest mismatch value (i.e. best image 

15 smoothness at K = K^est) is selected, at 120. Accordingly, 

the texture data of the first N-Kbest macroblocks is restored 
from the temporary buffer created in 108. The texture data 
of these N-Ki,sst macroblocks is the recovered texture data. 
The last Ktest macroblocks then include concealed texture 

2 0 data. Finally, the processing moves to the next video 
packet, at 122. 

FIG. 3 illustrates an image smoothness test of texture 
data in accordance with an embodiment of the present 
invention. The image smoothness of the current macroblock 
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3 00 may be evaluated by summing the square of the pixel value 
mismatch between the current macroblock 300 and the 
neighboring macroblock 302, 304, 306, 308 boundary pixels. 
However, other methods, such as root -sum- square or simple 
5 difference sum, may be employed to compute the boundary pixel 
mismatch value. Further, the image smoothness test may be 
performed with more than one row or column difference of the 
boundary pixels between neighboring macroblocks . 

In the illustrated embodiment, the boundary pixel 
10 mismatch value of the current macroblock 300 is computed 



Al = Yj^a ■(^O.i - ^I5,,f + Wft -(^IS,,- -So,if+^l -(^,,0 -Li,i5f + -(^,,15 -%)^ • 

The weights Wa, w^, Wi, and Wr may be used to weigh the 
boundary mismatch between macroblocks belonging to different 
15 video packets differently. Thus, depending on the desired 
quality of video frames, the boundary mismatch between 
macroblocks that belong to different video packets may be 
configured to weigh more than the mismatch between the 
macroblocks that belong to same video packets . 

2 0 An error concealment system 4 00 according to an 

embodiment of the present invention is shown in FIG. 4. The 
system 400 includes an error location detector 402 to receive 
11 
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N macroblocks of texture data, some of which are corrupted. 
In the illustrated embodiment, the detector 402 is configured 
to determine the location of a video packet error with 
respect to the total number of macroblocks (N) . The location 
5 determination provides the macroblock number {K±nit) from the 
end 210 of the texture partition. Thus, this determination 
enables separation of N macroblocks of texture data into N-K 
recovered macroblocks and K concealed macroblocks. 

The N-Kinit macroblocks are stored in a storage element 
10 404. The first N-Kbest macroblocks of the N-Kinit macroblocks 
are later used, once the value for Kbest is deterroined. The 
last Kinit macroblocks of the texture data are then sent to an 
error concealment element 406. In one embodiment, the error 
concealment element 406 includes a motion compensated 
15 temporal replacement element. The temporal replacement 

element uses motion information to propagate macroblocks of 
the texture data from the previous frame to replace the 
texture data of the macroblocks in the current frame . The 
first N-K macroblocks assumed to have non-corrupted texture 
20 information may be considered as "non- corrupted" in the above 
temporal replacement process. 

An image smoothness evaluator 408 then evaluates the 
image smoothness of the temporally replaced texture data. In 
the illustrated embodiment, the evaluator 408 measures the 
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smoothness of macroblock boundaries in the restored texture 
data. Particular implementations of the macroblock boundary 
test have been described above. 

A best image smoothness selector 410 receives a pixel 
5 mismatch value from the evaluator 40 8, and selects texture 
data that provides best image smoothness. This selection 
provides a value for Ktest- The selector 410 provides this 
value to the error concealment element 40 6 and the storage 
element 404. In response, the storage element 404 provides 
10 N-Kbest recovered decoded macroblocks, and the error 

concealment element 406 provides Kbest concealed macroblocks. 
These macroblocks of texture data may be combined to produce 
a best set of macroblocks of texture data to replace the 
corrupted video packet . 

15 There has been disclosed herein embodiments for error 

concealment of video data using texture data recovery. In 
one embodiment, the corrupted macroblocks may be concealed by 
motion compensated temporal replacement. To find an 
appropriate number of macroblocks to conceal, macroblock 

2 0 boundary smoothness is evaluated for varying number of 

macroblocks of utilized texture information. Thus, the final 
number of macroblocks of utilized decoded texture information 
(the recovered information) is selected to maximize overall 
boundary smoothness. 

13 



080398 .P482 



While specific einbodiments of the invention have been 
illustrated and described, such descriptions have been for 
purposes of illustration only and not by way of limitation. 
Accordingly, throughout this detailed description, for the 
5 purposes of explanation, numerous specific details were set 
forth in order to provide a thorough understanding of the 
present invention. It will be apparent, however, to one 
skilled in the art that the system and method may be 
practiced without some of these specific details. For 

10 example, error concealment may use a process other than the 
motion compensated temporal replacement. Moreover, the 
present embodiments may be used to process video data having 
a texture data unit other than a macroblock. Further, the 
methods of the present embodiments may be programmed as a 

15 software on a computer readable medium, such a floppy disk or 
a CD-ROM disk. In other instances, well-known structures and 
functions were not described in elaborate detail in order to 
avoid obscuring the subject matter of the present invention. 
Accordingly, the scope and spirit of the invention should be 

20 judged in terms of the claims which follow. 
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